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ABSTRACT 

The U.S. Department of Education ' s (ED) Special 
Services for Disadvantaged Students program was reviewed during 
1977-1980 using a sample of 11 grantee institutions. Attention was 
directed XQ the management of the program, whether program goals were 
being met, and progress of students in the program. The program's 
goal is to increase postsecondary retention and graduation rates of 
disadvantaged students. It is concluded that objectives of projects 
are not always consistent with the program goal, and ED has not 
adequately monitored projects to determine whether the goal or 
objectives have been met. Four of the 11 projects did not have 
objectives concerning student retention rates or academic 
achievement. Although a primary means of assessing project 
performance is site visits, in fiscal year 1980, ED visited only 24 
of the -557 grantees. A second main measure of project performance is 
annual reporting,- which. did not compare actual performance with 
proposed objectives. Recommendations address the need for measurable 
objectives, visits to monitor the attainment of ^objectives, and 
reporting requirements. Appended materials identify the institutions 
that, were reviewed, and include data on student tutoring and 
counseling' services, along with a response to the review by ED. 
(SW) * . 
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BY THE US. 

Report To The Chairman, Committee 
On Labor And Human Resources 
United States Senate 
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About Effectiveness Of Its Special 
Services Program 



o 



About $60 million a year is spent to fund, 
about 600 projects under the Special Serv- 
ices for Disadvantaged Students program. 
The Department of Education has not ade- 
quately monitored the projects to deter- 
mine whether their goal of increasing the 
postsecondary retention and graduation 
rates of disadvantaged students has been 
achieved. 
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UNITED STATES GENERAL ACCOUNTING OFFICE 
WASHINGTON, D.C. 20648 



HUMAN RESOURCES 
DIVISION 



B-209515 



The Honorable Orrin, G. Hatch 
Chairman* Committee on Labor and 

Human Resource! 
United States Senate 

Dear Mr. Chairman i 

In vour letter of February 20, 19817 you requested that 
we review the Department of Education' s Special Programs for 
Students from Disadvantaged Backgrounds- commonly known as 
the TRIO programs. We have reviewed the administration of two 
TRIO Droarams— Special Services for Disadvantaged Students and 
UpwarS S! tls report concerns ^e Special Services program, 
a separate report on the- Upward Bound program will be issued to 
you at a later date. 

This report contains recommendations to the Secretary of 
^Educafion wSfcS sSoSld assist him in determining if f« Prpg ram' s 
goal and the objectives of individual projects funded by this 
program are being met. 

As arranged with your office, we will not^ release the report 
for 30 days unless youSpprove its release or 

nublic. At that time, we will send copies to the Secretary or 
lduca?ion, the Director of the Office of Management and Budget, 
the Special Services grantees included i^ the study, and other 
interested parties r we will also make copies available to others 
upon request.. 

Sincerely yours. 



0L a^Mu^k 

Philip A. Bernstein 
Director 
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GENERAL ACCOUNTING OFFICE 
REPORT TO THE CHAIRMAN, 
COMMITTEE ON LABOR AND 
HUMAN RESOURCES 
UNITED STATES SENATE 



DEPARTMENT OF EDUCATION 
UNCERTAIN ABOUT EFFECTIVENESS 
OF ITS SPECIAL SERVICES PROGRAM 



D I_ G E S_ 'T 

The Department of Education (ED) does not have 
assurance that the Special Se rvices for Di£ 
advantaged Students program is achieving its 
goal of increasing the retention and graduation 
rates of disadvantaged postsecondary /tudents. 
Under the program, (grants are awarded to post- 
secondary institutions supporting projects^to 
assist disadvantaged students to successfully 
pursue their programs. of study by Providing them 
with services, such ajp tutoring, counseling, and 
special classes. '.. 

Prolects' objectives are not always consi » ten ^ a - 
wi?h the program's goal, and ED has not adequately 
^oStoredVojects to determine Aether the goal 
or the objectives have been met. Four of tne 
11 projects GAO visited did not have objectives 
concerning student retention rates or academic 
achievements. (See pp. 5 and 6.) 

to assess the performance of a project, ED relies 
primarily on site visits and project-prepared per- 
' rormanc^reports. GAO fbund, however, that the 
visits and Reports did not provide adequate data 
to assess performance. In fiscal year 1980, ED 
made site visits to only 24 of the 557 grantees, 
during which, ED did not determine whether pro j- 
ects were meeting their objectives and did little 
followup to assure that any noted deficiencies • 
were corrected. GAO did note instances where pro- 
posed objectives were not achieved, and there wae 
no mention of this in ED's reports on visits to. 
these sites,. (See pp. 7 and 8.) 

Although ED requires projects to report annually^ 
on the status of their operations, the reports did 
not compare actual performance with Proposed ob- 
jectives. At the projects visited, GAO noted^ 
'several instances where objectives were not ac 
complished or measured and these were not re- 
ported in performance reports. (See p. 8.) 
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The performance reports also do not provide informa- 
tion on the academic. status of the Special Services 
students. Such information would be useful in assess- 
ing how well the program's retention and graduation 
goal is being met and in identifing projects whose 
methods have been successful in accomplishing the 
goal. For example, the participants at one project, 
which requires students to attend project-sponsored 
clasXes full time during their freshman year, had a 
retention rate of 98 percent as compared to the in- 
stitution 1 s overall retention rate of 76 percent. 
(See pp. 9 and 10. ) 

RECOMMENDATIONS TO THE 
SECRETARY OF EDUCATION 

To better determine whether the program goal and proj- 
ect, objectives are being met, the Secretary should 
require proposals to contain objectives consistent 
with the program goal of increasing retention and 
graduation rates, (1) site visits to consider whether 
projects are accomplishing proposed objectives and 
(2) projects to report the status of all objectives 
in their performance reports. (See p. 10.) 

DEPARTMENT OF EDUCATION 
COMMENTS AND GAP'S EVALUATION 

ED concurred with GAO's recommendations stating' 
that a need exists to improve the means by which 
the project objectives a$d overall program goal 
of several aspects of the Special Services for Dis- 
advantaged Students Programs are being met. 

However, ED did not agree that it does not know 
whether the Special Services for Disadvantaged Stu- 
dents Program is achieving its goal. ED contends 
there is substantial evidence that indicates the 
program is achieving the program goal. To support 
its contention ED cited data from three sources. 
However, GAO does not believe that these sources pro- 
vide ED with sufficient proof. (See pp. 10 to 13.) 

SCOPE AND METHODOLOGY 

GAO undertook this review to (1) assess how well ED , 
managed the Special Services program, (2) determine 
if projects were meeting project and program goals, 
and (3) determine the progress of students in the 
program. GAO reviewed a 3-year period of the program 
(1977-80) at 11 judgmentally selected grantees. 
Therefore, GAO cannot project its findings to all 557 
projects operating , during fiscal year 1980. (See pp. 
3 and 4. ) 
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CHAPTER 1 



INTRODUCTION 



The Department of Education's (ED's) Special Services for 
Disadvantaged Students program assists disadvantaged P o ;tsecond- 
ary students to- pursue their programs of study by providing them 
with supportive services, such as tutoring, counseling, and special 
academic classes. During the program years we examined ( 1977-/b v . 
through 1979-80), eligible participants were postsecondary stu- 
dents who needed remedial services because they were educationally, 
culturally? or economically deprived, had limited English •P-aking 
ability, or were physically handicapped. The 1980 amendments to 
the Higher Education Act of 1965 changed the eligibility require- 
ments. Now a participant must be a low-income, physical ly handi- 
capped, or first-generation college student (a person neither of 
whose parents has completed a baccalaureate degree). \ 

Although the authorizing legislation does not specifically 
identify the goal of the Special Services program, the pr°9 ram 
regulations in effect at the time of our review stated that the 
program's goal was to increase the postsecondary retention and 
' graduation rates of participants. The recently issued Special 
Services regulations, implementing the 1980 amendments to the 
Higher Education Act, do not include a statement of the program a 
goal. However, the program was proposed to the Congresses one 
which would assist disadvantaged students in completing their 
undergraduate education, and the legislative history of the 1980 
amendments does not indicate that the Congress intended to change 
the program's goal. Therefore, we believe postsecondary retention 
and graduation of participants is still the appropriate program 
goal . 

The Special Services program was authorized by the Education 
Amendments of 1968 (Pub. L. No. 90-575) to Title IV-A Subpart 4 
of the Higher Education Act of 1965, as amended (20 U.S.C. 1070.). 
The program is one of four programs under Title IV-A aimed at re- 
moving nonfinancial barriers to entering and completing post- 
secondary education far students from disadvantaged backgrounds. 
Those programs (sometimes collectively referred to as TRIO pro- 
grams) are intended to complement other programs in title IV, 
which offer student financial assistance. 

* The Special Services program received its initial appropria- 

tion in 1970. Through fiscal year 1982, the program had been ap- 
propriated about $447 million. In fiscal year 1982, the program 
received $61 million which supported' pro jects at 640 postsecondary 
institutions. A typical Special Services project at a postsecond- 
ary institution is supported by a $95, 000 grant and is funded to 
serve an average of about 23 5 students. 
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PROGRAM ADMINISTRATION 



ED 1 s Office of Postsecondary Education, Office of Institu- 
tional Support Programs, Division of Student Services, manages the 
Special Services program from ED's headquarters in Washington, D.C. 
This office develops policies for program operation, reviews- and 
approves project proposals, monitors and provides technical assist- 
ance to grantees, and evaluates pro jects * effectiveness. 

The Special Services program is a discretionary grant program 
for which institutions of higher education . must compete for avail- 
able funds. To obtain funds, prospective grantees submit proposals 
that outline their plans for providing services to assist eligible 
students to initiate, continue, and complete postsecondary educa- 
tion. ED employs field readers 1/ to evaluate and score proposals 
based on factors, such as the need for services, the soundness of 
the project's design to provide the services, the adequacy of, the 
applicant's resources and organization and the project's estimated 
cost. In addition, recently issued regulations , implementing a 
provision of the 1980 amendments to the Higher Education Act, re- 
quire that an applicant's prior experience in the program be con- 
sidered when Awarding new grants. 

After field readers evaluate the proposals ,/ ED officials re- 
view the readers' recommendations and determine how many projects 
will be funded and the funding level of each. ED usually funds 
projects for a 3-year period, with the second and third years being 
funded as lone) as funds are available and the need still exists. 
When the multiyear funding cycle ends, projects must develop new , 
* proposals and again participate in the competitive funding process. 

PROJECT OPERATION 

Special Services projects operate under broad ED regulations 
and consequently vary in their methods of providing services. For 
instance, in selecting participants, some projects actively seek 
' to identify eligible students before their enrollment in school 
and encourage them to use program services. Other projects re- 
quire students to participate if they -do not meet regular admis- 
sion standards. Still others may publicize their services but 
leave it tp the students to seek the services or rely on instruc- 
tors and/or counselors to request that students obtain the serv- 
ices. 

The projects also vary in the mix and extent of services 
offered. Most projects offer tutoring and counseling services, 
and some also offer project-sponsored remedial classes. Most 



l/Non-Federal reviewers who have expertise in educating dis- 
advantaged persons . 
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projects we visited design their services^ meet the need, of 
rr.shmen. However, some proje ct. "provHeS Student. 

S^STiJK-JSroS'^SSlSS ~ ....ion to s 

full year's course plus tutoring and counseling. 

OBJECTIVES, SCOPE, AND METHODOLOGY 

Our review was made in accordance with generally accepted 
our " V "" 4 . W " J| , .L. It focused on ED ' s management of the 
government audit standards. it rocuaeu u » 
qnecial Services program. The purpose of the review was to n 
assess how well ED managed the Special Services program, (2) d« 
e i? the projects were meeting project and program goals 
"J" ! determine the progress of students in the program. Aft 
J*!*«ft£e review, the Chairman of the Senate Committee on 
^oranrHu^n^Re^^cerexpressed an interest in the assignment 
and asked that the report be issued to .his Committee. 

Our review covered program operations in fiscal years 1978, 
1979 and?980 ( academic years 1977-78, 1978-79, and ^79-80). 
lll»Am the review at ED headquarters in Washington, D.C., and at 
J l^tfl^ie.. promts. ^ ED head garters, we in- 
. j ' ^ui. 4 ri 4-vie Division of Student. wrvicBo wuu °^ 
t rTi?«f t SrSDecial Services program. The interviews focused on 
tie ouKance Id provides to grantees and the methods ED use. to 

ices iegiBiatiun, a/q/Mf 4 nn we reviewed a report 1/ on 

SS-iM! SSKi- oV^ ^ en ? 0 

^•thr^^^s^^r:tS.:s: received 

services during that period. 

. fhorp werp a total of 557 grantees in tne bpeciaj. 

S^^^ r -£Wi5iSi SSxfi » £ v.te, . 

institutions that were awarded grants covering the 3 -year period 

^ c „.« _ i c Brv icBB for Disadvantaged Students 

Corporation, August 1981 (TM-6 198/003 /00 ) . 
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beginning with academic year 1977-78.. The institutions selected • 
received grant fundB totaling about *$3. 0 , million during the 3-year 
period. Program officials responsible for monitoring Special Serv- 
ices grantees told us they believed we had chosen a representative 
sample. , ' 

At each institution, we conducted structured interviews with 
project officials and reviewed the approved project proposals, per- 
formance reports, financial records, student transcripts, and, other 
data that provided insight on project operations. We compared each 
project's accomplishments with the objectives outlined in its pro- 
posal. We selected a simple random sample of. students from project 
rosters by year for each year of the 3-year funding cycle and at- 
tempted to determine 

— if they met the program's eligibility criteria; 
— the extent and frequency of services received; 

— their progress in terms of credit hours attempted and passed; 

— their c)rade point average; and 

— if they were still enrolled, withdrew, or graduated. 

Our sample included 707 students or about 10. percent of those who 
were supposed to have participated at the projects selected. 

Because the projects we visited were judgmentally selected 
and the number was limited due to time and staff constraints, we 
could not project, with any statistical validity , our findings 
regarding program performance. However, our findings at these 
11 projects indicate a need for increased attention by ED manage- 
ment concerning the program's administration-. 



/ CHAPTER 2 

ED NEEDS BETTER ASSURANCE THAT T HE PROGRAM QOAL 

/ AND PROJECT OBJECTIV gfl ARE BEING MET 

ED do*, not haVe adequate assurance that tht Special Service, 
program is achieving it. goal of increasing the rjtjntion * nd _ 
graduation rate, of disadvantaged students in PO«t«econdary .•*uca 
?<«n Pro-Uet.' obi.ctiv.s are not always consistent with the pro- 
gram's goal ED Sa. not adequately P»j«oti to djt««ln« 
ShSthlr thi program goaf or the stated objectives of the projects 
hav* be*n mmt. 

To assess the projects' performance, ED relies primarily on 
information collected by it. program monitor, during Project site 
visit, and information contained in performance "Ports submitted 
fay the grantee.. Neither the vi.it. nor the reports have Provided 
2 Senate basis to asses, performance. ED ha. mad. visit, 
to the project, covered by our review, and the visits 
made did not measure the projects' performance gainst th«ir_ apr 
proved objectives. Some grantee performance report, have not con 
tained complete information concerning project activitle.. <fjr ^ 

PROJECT OBJECTIVES NOT ALWAYS / 
CONSISTENT WITH THE PROGRAM GOAL 

Even though increased retention and 9^duation of disadvan- 
taged students in postsecondary education is the Priory goal pf . 
Special Services, ED has not required projects to """■"J 
objectives for participants' postsecondary retention or academic 
achievement 8. Only 3 of the 11 projects we visited had an objec 
tive specifically stated in terms of retention and graduation or 
students. 

— A public 2-year college proposed that 180 of the approxi- 
mately 800 participants served during the 3-year grant 

. period 1977-80 would complete their program and graduate 
with an Associate Degree or Certificate; 154 of the 
project's participants received their degrees during this 
period* 

—A public university proposed to increase its Participating 
students' retention rate to 80 percent, the parti chants 
retention rate over the proposal period (1977-80) averaged 
98 percent. 

— A private 4-year college proposed that 60 percent of the 
participants' would graduate, the Project, however, did not 
document its performance against this objective. 
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Six of the 11 pro jects, including 2 of the above, had measurable 
objectives for student achievements stated in terms of courses 
passed, grade point averages, or scores on standardized tests . 
Generally, however, the projects did not achieve the objectives 
or did not determine if these objectives were achieved. 

Usually projects had objectives relating to the. number of 
students to be served and the services to be provided. Most other 
objectives, however , were stated in broad, vague terms that were 
not ^measurable . For example: ' 

— One project proposed to "create a high level of knowledge 
about the program o n- campus . " Another objective stated 
that "each individual will be knowledgeable and effective 
in utilizing college procedures and policies pertaining to 
, academic and career development." 

7 --Another project proposed "to provide daily incentives 

(motivators) , appropriate cultural enrichment and remedial 
classes/activities, as approved.by the program director, 
by program staff to improve a total commitment to academic 
and personal- social excellency for each eligible student ." 

—A major Objective of one project was "to provide culturally 
deprived students with opportunities to remedy deficiencies 
' . . ' in their ability to understand, appreciate , and relate to 
« the dominant cultural patterns and expressions in their 
environment." 

Although these may be valid objectives, their relationship to the 
program goal concerning student retention and graduation rates was 
not indicated in the granjt proposals. 

On June 8, 1982, ED issued program regulations which require 
that an applicant's prior experience in the Special Services pro- 
gram be considered when awarding new grants. These regulations 
consider the extent to which project participants persisted ( in- 
cludes both retention and graduation) toward completing their 
academic programs. However, they do not require projects to set- 
objectives, related to increasing participants 1 persistence or 
performance. 

ASSESSMENTS NOT MADE TO DETERMINE 

WHETHER PROJECT OBJECTIVES ARE MET ' 'J y 

ED has relied primarily on site visits and' grantee-prepared 
performance reports to assess the projects 1 performance. Under 
its new regulations,, ED plans to rely on these visits and reports 
as well as audit reports and information in thff proposals, to 
assess a project's prior experience. A program official told us 
that audit reports are of limited assistance in assessing a proj- 
ect 1 s performance because they are often'not timely and dp not 
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.consider many important aspects of the program* None of the 
grantees we visited had audits that determined if the project 

.achieved the program goal and project objectives. Site visits ( 
have been infrequent and limited in scope, and performance repoVts 
have not always been complete or accurate ; therefore^, neither 8 
assessment approach has provided an adequate basis for assessing^ , 
a project's performance. . 

\ , -.' ■ • '•• •• : '' 

Site visits are infrequent 

a nd limited in^ scope 
r~ — — — — : — ! — 

During fiscal year 1980 (academic year 1979-80), ED made 
only 24 site visits to the '557 Special Services projects. Of the 
11 projects we reviewed-, 3 were visited once during the 3-year 
grant period (1977-80); 3 were not visited in abou^t the last 
5 years; and according to project officials, 5 had never been 
visited since -receiving their initial Special Services grant. 
Beginning in fiscal year 1981, ED program officials developed 
guidelines requiring one- fourth of the projects to be visited each 
year. However, ED, officials stated that, due to limited staff and 
severe travel "cutbacks, they have not been able tb meet this 
guideline . . ' 

The primary ob jectives of -ED site visits are to determine 
whether projects are complying with program regulations arid ac- 
complishing approved proposed objectives. During a site visit, ED 
monitors use a pro forma questionnaire to obtain information about 
the project and compliance with documentation requirements. The 
questionnaire, however, does not address matters specifically 
related to implementation of proposed objectives. Consequently, 
during its site visits, ED personnel generally did not determine 
whether the projects' proposed objectives were being met* 

All three projects we visited that had a site visit during 
the 3-year grant period (1977-80) failed to meet one or more of 
their objectives. One project did not accomplish 5 of its 10 ob- 
jectives; another project did not accomplish 1 of 3 objectives. 
The third project proposed six objectives/ however, two were in 
broad^L general terms that .could not be measured, and three of the 
four measurable objectives were not achieved. The site visit re- 
port for these projects gave no indication that proposed objec- 
tives were not being met.. 

£ite visits were not as effective as they could have been 
because, even'when deficiencies were identified, ED did riot always 
follow up to determine if the projects corrected the deficiencies 
noted during the visits. For example, a site visit report on a 
project noted- a number of deficiencies, including 

— no documentation of counseling services,. 

— a^lack of documentation of student eligibility, and 
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r— inadequate tracking of student© leaving the program. 

Our review pf this project, about 3 yeats after the site visit, 
showed a continued laqk of documentation of eligibility for a 
significant number, of participants, no documentation 6f counsel- 
ing services, and no -attempts to track participants leaving the 
program. The project director stated that ED had not contacted 
the project to determine if the deficiencies were corrected. 

After completion of our review, new procedure* for 0 monitor- 
ing Visits to projects were developed- The pro forma question- 
naires used with these new procedures does not specifically require , 
a comparison of project objectives and accomplishments. 

■ • •* * 

Project reports need to be 
complete £nd accurate ; 

At the time of our review, ED required Special Services proj- 
ects to submit reports of their performance semiannually.. Now such 
reports are required annually. The reports, which are intended to 
assist ED in. its monitoring of project operations, include such, 
information as: ~ ' 

— The number of participants served. 

— The basis for each participant's eligibility for services. 

— The participants' distribution by sex and ethno- racial 
background* 

-t-The number of students receiving project services such as 
counseling, tutoring, orientation, and classroom instruction 

— The reasons participants left the project. 

In addition, projects may write a brief narrative comparing accom- 
plishments with their proposed objectives. ED requires this in- 
formation, however, only "if the objectives were not attained. 

At the projects visited, 'we compared performance against 
proposed objectives and noted instances where objectives were not 
accomplished or measured, but this was not noted in performance 
reports. For example, five of the projects failed to report that 
they fell short of serving the number of students proposed. One 
project proposed to raise the reading level of at least 75 percent 
of its participants to the 50th percentile by the end of 1 academic 
year; however, it did not measure participant achievement levels. 
Another" project proposed to provide services that would enable at , 
least 7 5 percent of its participants to, complete required courses; 
however, the project did not track students- to determine, if they 
successfully completed courses. x 
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The reports also did not show that, at some projects, stu- 
dents were not using tutoring and counseling services to the degree 
proposed. Although seven of the projects we visited proposed to 
provide counselin#to each participant, only 265 (58 percent) of 
the sample participants at" these projects received this service.^ 
(See app. III.) Six projects proposed to provide tutoring services 
to each participant. One of these projects did not document the 
provision of tutoring services, and only L53 (42 percent) of the 
sampled students at the other five projects received any tutoring. 
(See app. II.) At a project which proposed to provide individual- 
ized.-tutoring to those st'udents needing the service, the , 25 sampled 
participants failed to attend 77 percent of a their scheduled tutor- 
ing sessions. The reports of these prbjeqts did not state that 
tutoring and counseling services were not being used as proposed. 

The project reports are not required to provide information 
on the academic status of Special Services students. Such in-' . 
formation would be useful in assessing how well the program s goal 
of increased student retention and graduation is being met. It 



would also help to identify projects that might be considered ex- 
emplary and whose methods, if adopted, may be helpful to other 
projects. In this regard, the progress of freshmen students in one 
project was impressive in terms of retention and academic per- 
formance. The only difference between this project and others we 
visited was that, in addition to tutoring, counseling, and other 
special services, students attended pro ject- sponsored credit 
courses on a full-time basis during their freshmen year. The proj- 
ect's records showed that about 98 percent of the Participating 
students who entered school during academic years 1977-7 8, 1978-7 9, 
and 1979-80 enrolled the following year. .This compares to the 
institution's average retention rate for freshmen who entered dur- 
ing these same academic years of 76 percent. In addition, from a 
random sample of -38 students who participated in the project during 
1977-80x^30 (79 percent) had overall grade point averages of 2.0 
or better, and only 6 (16 percent) eventually withdrew from school. 
Of the 32 students who did not withdraw, only 6 (19 percent) were 
2 or more terms behind what the institution considered the normal 
rate of progression toward graduation (!..•.# 2 years in 2-year 
school, 4 years in 4-year school) . 

We compared the characteristics (i.e., reasons eligible, sex, 
ethno-racial background, etc.) of the students participating in 
this project with those of other projects visited and found no' 
appreciable differences. In addition, there was nothing especially 
unique about/the institution hosting the project. However; by com- 
parison at the other projects we visited, only 59 percent (277 of 
471 students) of the sampled students had overall grade point aver- 
ages of 2.0 or better, and about 50 percent (276 of 552 students) 
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had withdrawn frpm school. 1/ Of those students who had not with- 
drawn, about 3 3 percent were 2 or more terms behind the normal rate 
of progression toward graduation. 

conclusions •. * 

Based on our review of ED' s policies and procedures for moni- 
toring the Special Services projects and our test of these proce- 
dures at a small sample of projects, we believe ED has little basis 
to determine if the Special Services program is meeting its goal 
of increasing the postsecondary retention and graduation rates of 
disadvantaged students. Requiring projects to set measurable ob- 
jectives consistent with the program goal and requiring projects 
to report on the academic performance of participants and the 
status of other project objectives would help to assess the Pro- 
gram's achievements. Also, expanding monitoring visits so that, 
they include a review , of whether the program goal and project ob- 
jectives are being achieved would also help ED assess the program. 

RECOMMENDATIONS - 

To better determine whether the program goal and project ob- 
jectives are being met, the Secretary of Education should require 

(1) project proposals to contain measurable objectives consistent 
with the program goal of increasing retention and graduation rates, 

(2) project monitoring visits to determine whether projects are 
meeting proposed objectives, and (3) project annual performance 
reports to include information on the academic performance of par- 
ticipating students and the status of all proposed objectives . 
whether accomplished or not. 

AGENCY COMMENTS AND OUR EVALUATION 

In commenting on a draft of our report (see app. IV), ED con- 
curred with our recommendations. ED stated that a need exists to 
improve the means by which the project objectives and ^overall pro- 
gram goal of several aspects of the Special Services for Dis- 
advantaged Students Program are being met.' 

However, ED did not agree with the main message of our report 
that it does not know whether the Special Services for Disadvan- 
taged Students Program is achieving its goal. ED contends there 
is substantial evidence from several sources that indicates the 
program is achieving its goal. To support its contention ED 



1/Grade point averages could not be computed for\81 of the sampled 



students. j 
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cited data from three sources. We do not believe^ these data^suf- . 
ficiently support ED's contention, and continue tio believe that 
there is dSnsiderable uncertainty as to whether t^he program goal 
is being achieved. 

The first source of data cited by ED is a" compilation of 
annual performance reports submitted by Special Services grantees 
for program years 1978-79 through 1980-81. .(Our review covered 
program years 1977-78 through 1979-80 . ) Thefce data show that an . 
average of 67 percent of. the students served each year remained 
enrolled in the project and in school throughout the year . . Ac- 
cording to the data, about 20 percent of the students were no 
longer enrolled in the project because they graduated from college 
(5.2 percent), transferred to another school (2.4 percent), or no 
longer needed the services of the project (12.5 percent). The 
remainder of about 13 percent dropped out of the program for var- 
ious-reasons, including personal, health, and academic dismissals. 

■ ' . 

While the performance reports indicate that 67 percent of the 
students remained in the project, they do not indicate whether 
these students were satisfactorily progressing toward graduation— 
the ultimate goal of the program. At the 11 pro 3 ects we visited, 
many of the students on whom we obtained data were not J**^. 
standards of academic progress and performance established by their 
schools. Forty percent of the students sampled had grade point 
averages below 2.0 (the minimum required to meet graduation re- 
quirements), 48 percent had dropped out of school , an<T61 P«9J"J ■ 
of those remaining in school were not progressing toward graduation 
at what their schools, considered a normal rate . 1/ The retention 
of some students may be partially attributed to program regulations 
in effect during the period covered by the data that required 
schools to retain Special Services students for a minimum time per- 
iod 2/ regardless of their academic performance. 

. A second source ED used to support its contention was data 
obtained from a recent initiative undertaken by ED 's Of f ice of 
Postsecondary Education. This initiative was designed to identify 
and disseminate information on programs and practices ^exemplary 
quality which were cost effective. According to officials in ED s 
Office of Postsecondary Education, under this initiative all 6 1J 

1/Our sample was composed of a total of 707 students at the 11 
" projects. However, due to factors such. as unavailable tran- 
scripts, students withdrawing from all courses, and programs 
which did not record Wades of less than a "e," ou ^ am P^° r 
grade point averages aiid rates of progression was 509 and our 
sample for drop out rates was 584. 

2/The minimum retention periods were 1 year in a 2-year school 
and 2 years in a 4-year school. This provision was, eliminated 
from revised regulations issued in March 1982. 
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Special Services projects were invited to submit data on their 
operations if they thought the project was exemplary or used* ex- 
emplary practices. Approximately 35 projects submitted such data. 
Ah independent panel of ED personnel determined that 19 projects 
had submitted data which described programs or practices which were 
cost effective and of exemplary quality • The 19 projects represent 
less than 4 percent of the 613 projects and would not, in our opin- 
ion, provide a sufficient basis for determining whether the program 
as a whole is accomplishing its goal. 

The third source of data which ED cited was a contracted study 
of the Special Services program conducted during the 1979-80 aca- 
demic year by System Development Corporation. Among >other things „ 
the study examined (1) the effect of the support services provided 
by the Special Services projects and by other sources on the re- 
tention rates of a student sample composed of Special Services par- 
ticipants and nonparticipants with characteristics similar to the 
participants and (2) the amount and type of support services pro- 
vided by the sampled Special Services projects to their partici- 
pants . 

The study divided the student sample into 11 groups depending «. 
on which services they received and whether the Special Services 
project or some other source, provided each of the services. These 
groups ranged from one which received no support services to one 
that received the full range of services (tutoring, counseling, 
and group instruction) from the Special Services projects to some 
that received support services only from sources other t^ari th;© ;; 
Special Services pro jects . Five of the 10 groups which deceived >< 
services had no greater odds of persisting through their freshman 
year than students who received no support services. The other 
five groups had greater odds of persisting. Only one of these 
five received all of its suppprt "services from the Special Servr 
ices project. This group received the full range of services and 
had the greatest increased odds (2.26) of persistence. While the 
report did npt state the number or percent of Special Services 
students in the sample who received this full range of services 
from the Special Services projects, ED officials, after contacting 
the study 1 s project officer, advised us that only about 12 percent 
of the sampled Special Services students were in this group. 

The study report also noted that students 1 persistence in 
academic studies and the number of credits attempted and earned 
are related to the amount of services received. However, when 
examining the services provided by the sampled Special Services 
projects to all of their participants, the study found that about 
half of the Special Services students received no tutoring, about 
two-thirds received no group instruction, and about one-third re- 
ceived no counseling. Our review also showed that Special Serv- 
ices participants made limited use of the services offered by the 
projects. Fprty-nine percept of the students in our sample re- 
ceived no tutoring and 42 percent received no counseling. We did 

* * 
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not Obtain data on the number of students participating in 
group instruction. , 

Accordingly, although the study shows a positive . ^J*»£ 
ship between student persistence and progress and the amount of 
services received, its data on the use of je^ices by Special 
Services participants indicate that a substantial portion of 
Special Services participants received limited services and might 
not be realizing ?he increased potential for retention indicated 
by the study. 

ED \s specific comments on each of our recommendations is 
presented below. 

Require project proposals to 
contain measurable objectives 
consistent with the programgoal 

ED said that it will instruct potential grantees, field 
riders and ED program officers of the need for Special Services 
exposals ?o contain measurable objectives related to the program 
£al? The g^ auch objectives will be assessed 

during monitoring activities and when evaluating a grantee s prior 
experience in connection with the award of a new grant. 

Project monitoring visits should 

determine whether projects are t 
meeting proposed objectives 

ED said that program officers will be specifically instructed 
to assess the goal and objectives that impact most directly upon 
p!r?"?panr?e!ention and'graduation . If these objectives are not 
being met, or if it is impossible to validate performance data, 
the program officer will be required to make recommendations to 
corVec? the problems. The site visit report will document the 
program officer's findings and any required followup. 

Require project performan ce reports to 
Inc lude information on the academic 
performance of participating students and 
the status of all proposed objectives 

ED said that it is developing a new Special Services Project 

performance report. This new report will "^"^igciDanJs 
on the academic performance and progress of project participants. 
ED officials said they would use the report both as a project _ 
monitoring device and as a means for aggregating overall student 
performance, retention, and graduation data for the purpose of 
assessing the program goal. 
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INSTITUTIONS VISITED DURING REVIEW AND 
AWARD AMOUNTS FOR THE 3 -YEAR PERIOD 1977-80 

4- year public ' 

Georgia State University, Atlanta, Georgia $223,448 

Pan American University, Edihburg, Texas 515,437 

Rhode Island College, Providence, Rhode Island 299,862 
Southeastern Massachusetts University, 

North Dartmouth, Massachusetts 289,738 
University of/ South Carolina, Columbia, ' 

South Carolina 257,059 

4-year private 

Clark College, Atlanta, Georgia 219,398 

Huston-Til lotson College, Austin, Texas 170,013 

Morris Brown College, Atlanta, Georgia 205,517 

Saint Edward's University, Austin, Texas 260,403 



2-year public 



Bristol Community College, Fall River, 

Massachusetts 346,888 
Lurleen B 7 . Wallace State Junior College, 

Andalusia, Alabama 203,363 

? $ 2,991,126 
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special Services students receiving tutoring 



Project 


Number of 
students 
sampled 


Number (percent) 
receivinq tutoring 


Number of participants 
to Whom the project pro- 
posed to provide tutoring 


A 


74 


65(88) 


Minimum of 80% 


B 


75 


29(39) 


All 


D 


75 


37(49) 


All 


F 


25 


16 (64) 


Unspecified 


G 


75 


7(9) 


All 


H 


62 


33 (53 ) 


All 


I 


75 ' 


47 (63 ) 


All 


Total 


a/461 


234 (51) 




Total for proj- 
ects proposing 
to provide 
tutoring to 
all par- 
ticipants 362 


153 (42 ) 





a/We sanpled 707 participants. However, project "C , " ^ch^prcposed to 
provide tutoring to all participants, and projects E, J, and 
»K, " which proposed to provide tutoring to an unspecified nuriber of 
participants, were not included in the appendix because they did not 
document tutoring services.. 



15 



22 



APPENDIX III 



APPENDIX III 



SPECIAL SERVICES STOPPHS RBCEIVIN3 COQNSELINB 

Nurtoer of Nuttosr of participants 

students Number (psrcent) to Whom project proposed 



Project 


•anplsd 


receiving counseling 


to provide oounatling 


A 


mm * 

74 


36 (49 ) 


Minimun of 50% " 


B 


75 


52 (70 ) 


All 


C 


72 


44(61) 


All 


D 


75 


* 

55(73) 


All 


E 


25 


22(88) 


UnsDScifisd 


p 


25 


23 (92) 


All 


Q 


75 


13(17) 


All 


H 


62 


54(87) 


All 


I 


75 


24(32) 


All 


J 


75 


45(60) 


Uhspecified 


Total 


a/633 


368(58) 




■total for 
projects 
proposing 
to provide 
counseling 
to all par- 
ticipanta 459 


265(58) 





^/Vfe sampled 707 participants. However, project "K," Which proposed to 
provide counseling to an unspecified number of participants, was not 
included in this appendix because it did not document counseling 
services . 
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UNITED STATES DEPARTMENT OF EDUCATION 
OFFICE OF THE ASSISTANT SECRETARY FOR POSTSECONDARY EDUCATION 



SEP | 5 1982 



Mr. Gregory J. Ahart 



Director 

Hunan Resources Division 
Uiited States General Accounting 
Office 

Washington, D.C. 2Q548 
Dear Mr. Aharts 

Attached is our response to the draft GAD report on the "Sp eci a l 
Services, for Ddsalvatitaged Students Program". If there are any 
questions concerning thia matter, please contact Dr. Richard L. 
Fair ley, Director, institutional Support Programs. His telephone 
mater is (202) 755-1254. 




400 MARYLAND AVE., S.W WASHINGTON, D C 20302' 
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APPENDIX IV APPENDIX IV 

Response to GAO Study Entitled 

"The Department of Education Does Not Know 
Whether tha Special Services for Disadvantaged 
Students Program 1s Achieving Its Goals" 

As Indicated by our comments to the GAO recommendations outlined below, 
the Department of Education concurs that a need exists to Improve the 
. means by which tha project objectives and overall program goals 
of several aspects of the Special Services for Disadvantaged Students Program , 
are being met. We respectfully take exception to the title of the 
GAO report, howevjsr, which, without qualification, states that the 
Department of Education does not know whether the Special Services for 
Disadvantaged Students Program Is achieving Its goals. In fact, there 
1s substantial evidence from several sources that Indicate that the 
program Is achieving program goals. Further, we do not concur with 
the report's suggestion that the sample used by GAO 1s representative, 
nor with the Inference that the study 1s an accurate characterization 
of all of the SSDS projects nationwide. 

Although the GAO does Indicate, 1n their outline of the scope and 
methodology of this report, that -only "11 Judgementally selected 
grantees" were studied and that "...GAO cannot project Its findings 
to all 557 projects." (sic) (p. 111 GAO Draft), they fall to remind 
the reader of this crucial limitation when expressing their conclusions. 
As a result 1t 1s easy for the reader to conclude that this study 
accurately describes all SSDS projects and that there 1s virtually no 
real Indication that the program 1s accomplishing Its mission. The 
evidence reveals otherwise. 

Table 1, summarizes existing performance data for the years of the GAO 
study. This data represents a compilation of Information submitted by 
SSDS projects at the end of each project year 1n their Annual Performance 
Reports (ED Form 1231) 

August 23, 1982 
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TAIU 1 

iids rztrotKAia won 

1978 - 1981 



n 1978 - 1979 Tt 1979 - 1980 



Total Mo. of Stuianta 
toroll«d to I?Dt 

MuaoM roi ie avihg 



S80t Bo Loagar laqutrad 
Trtntf •rr«d 
Graduated" 

PlQIDCUl Aid 

Entaxari Military 
Far aonal 
RaaXtb 
teach 

Acad— 1c Dlawlaaal 

Adalc. Dlamlaaal 

CoDtUma in fro) act 
OapcofluUa 

Ha— U E» FraJact 



147,648 (100X) 



1|,719 (11 .31) 

3,615 ( 2.4X) 

7.704 ( 3.2X) 

1.393 ( W 

293 ( .21) 

9,084 ( 6,2X) 

991 ( .71) 

80 ( .03X) 

"3.333 ( 2.4X) 

321 ( ,2X) 



1.930 ( 1.3X1 
101. §41 ( 6ttJ 



172.986 (100X) 



21,433 (12.4X) 

' 4,137 ( 2.4X) 

8,967 ( 3.2X) 

1.478 ( -9X) 

368 ( .2X) 

11,039 ( 6.3*) 

1.249 ( ..7X) 

88 ( .03X) 

4.204 ( 2.4X) 

494 ( .3X) 

2.743 ( L6X) 

114,744 ( MX) 



n 1980 - 1981 Avaraaa 1978-61 



172.071 (100S) 



23.432 (13. 6X) 

4,270 ( 2.3X) 

9.080 ( 3.3X) 

1,813 ( LIS) 

339 ( -2X) 

11.166 ( 6.5X) 

L3*fl ( .8X1 

113, ( .06X) 

4,790 ( 2,7X1 

644 ( .4S) 

3,382 ( L9X] 

ill AA3 ( 631] 



164 . 2 3 3 (laOD 



20.328 (12. 3X) 

4,0;4 ( 2.4X) 

8.384 ( 3.2X) 

1,363 ( 1.0X) 

334 ( .2X) 

10,429 ( 6.4X) 

1,199 < .7X) 

94 ( .03X) 

4,176 ( 2. 35) 

486. ( .3X) 

2,692 ( 1.6X) 



SOURCE: U.S. Department of Education Special Services Final 
Performance Report Program Year 1978-1979; 1979-198U, 
and 1980-1981. 



As indicated In the fourth column of TaWe 1.. the average number of 
students served each year during the^three year period was 164.235. 
Of this nurter an average of 67 percent (110.136). continued to be 
enrolled 1n college and 1n the project each year. Of those who left 
the progr* during the year, an' average of 5.2 percent graduated from 
college and 2.4 percent transferred to other .postsecondary Institutions. 
. An addlttonal 12.5 percent left because they no longer needed the 
services of the SSOS project although they continued to be 'enrolled 
1n college. Therefore, over 87 percent, on the average, were 
achieving the program goal of -retention 1n. and/or graduation froa 
postsecondary education." 
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The Department concludes that If 87 percent of the low Income, dlsad- * 
vantaged students served during the three year period are still persisting 
or have graduated from postsecondary education that this constitutes 
success. We believe that these data provide the Department of Education 
with substantial assurance that program goals are bring attained. 

While about 13 percent of the students enrolled 1n SSDS did leave the 
projects for other reasons, during this period, only 4.4 percent departed 
••• for reasons that might be Interpreted as having an Impact upon Special 
Services Program^oals -(e.g. .Academic Dismissal, 2.5 percent, Administrative 
Dismissal .3% and Project Dismissal 1.6%). In as much # as all other reasons 
for departing (e.g. health, personal, death, military, financial ) appear 
to be outside of the control of project staff, 1t would seem that such^ 
numbers have a neutral effect on project or program success. 

Another example of why we feel that In large part the program Is meeting the 
legislated goal comes from a recent Initiative undertaken by the Off 1ce of Post- 
secondary Education. This effort was designed to Identify and disseminate 
Information oh programs and practices of exemplary quality and which are cost- 
effective. More than 35 SSDS projects responded to a rigorous questional™ 1n 
which they were required to provide detailed evidence of their effectiveness. 
The evidence documented 1n the responses to this Initiative strongly suggests 
that program goals and objectives are being met for these Institutions. ^ 
Aside from the Internal sources which tend to confirm that SSDS project 
objectives and program goals are substantially being met, the Department ^ 
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has other indications that program goals are essentially on target as 
well. For example, the 6A0 Report alludes toj/n evaluation of the SSDS 
program conducted during the 1979-80 academe year by Systems Development 
Corporation. In that evaluation a nationally representative sample of 58 
projects^were studied 1n depth. The study Included a detailed questionnaire 
of up to 200 students at each site, «uid revealed that "...student 
persistence 1n academic studies, theEumber of course units attempted and 
the number of units completed..." werl all positively related to partici- 
pation In the SSDS project. The report concluded that ""students receiving 
the full range of SSDS services examined here had predicted v odds of 
persisting (laying enrolled through the freshman academic year) 2.26 
times the odds for students who received no such services." (SDC Report, 
August 1981; pp. 12-13). On-going phases of the same evaluation study 
w1ll/prov1de additional data on student persistence and academic 
.achievement In subsequent academic years. 

In conclusion, It Is the position of the Department of Education tha't 
there Is substantial evidence to Indicate that Special Services for 
Disadvantaged Students program goals are being attained. > We would not 
want the Congress to Infer, however, that the^GAO Draft report 1s without 
merit. As Indicated 1n pur response to the specific recommendations, 
the Department Intends to Incorporate these several recommendations Into 
our current procedures. 

We do believe that the title, the Cover Summary and the Digest are misleading. 
A more accurate title for Instance, might well be": "The Department of '.v. 
Education Needs to Improve Their Methods of Approving Project Objectives ■■<■ 
and Monitoring Achievement of Such Objectives." 
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Recommendation #1 
GAO Recommends: 

To better determine wftether project objectives and program goals 
are being met the Secretary should require project proposals to 
contain measurable objectives consistent with the program goal of 
Increasing retention and graduation rates. 

The Department of Education agrees that specific measurable objectives 
consistent with program goals of Increasing retention and graduation of 
SSDS enrolled students must be Included 1n every grant application. * 
The Department will Instruct Field Readers to assess the 
degree to which proposed objectives are attainable and measurable. In 
pre-appHcat1ort workshops applicants will be reminded to specify measurable 
objectives 1n their application. It 1s our Intention to review the Infor- 
mation given to Field Readers, as well as the Application Technical Review 
Form which they use to evaluate, proposals 1n order to make certain that 
they clearly understand the primary goal of the program anld to evaluate * 
each proposal accordingly. In addition, ED program officer* will be 



Instructed to carefully review applications during their prAgr am review 
of highly ranked proposals to ascertain "that measurable objectives 
which assess graduation and retention of SSDS students are present. 
After grant approval a work plan on each approved project will be 
prepared with each project objective spelled out for future use 1n 
monitoring activities. 

i 

v I . ' 

I - 

It should also be mentioned 'that the 1980 Amendments to the Higher 
Education Act require that an applicant's prior experlerj^f lit the Special 
Services program be considered when awarding new grants. Regulations 
have been Issued establishing criteria by which prior experience 1s 
evaluated. These criteria are 1n large part based on a recognition 
that projects which have been achieving their objectives should be 
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given a priority advantage over those. that have not been as successful. 
A review document, which must be completed by ED program officers, requires 
an assessment of the proposed objectives of each project. 

\ 

Recommendation #2 
GAD Recommends: 

To better determine whether the program goals and yprajfcct objectives 
are being met the Secretary should require project monitoring visits 
to determine whether projects are meeting proposed objectives. 

The Department of Education agrees that program monitoring s1te-v1s1ts 

should assess the degree to which a project has been successful 1n 

attaining proposed objectives. It 1s also true that 1n the past, program 

monitoring site-visits concentrated on "process" rather than on program 

■outcomes." This approach has been modified. Program officers will be 

specifically Instructed to assess those "goals and objectives that Impact 

most directly upon participant retention and upon graduation. Where 1t . 

appears that objectives are not being met, or where 1t 1s Impossible to 

validate performance data, the program officer will be required to 

determine why verifiable data are not available and make recommendations 

to correct the problem. The program officer's findings will be 

documented 1n a Report of Site Visit with any required followup carefully 

spelled out. * 



Recommendation #3 

s 

GAD Recommends: 

To better determine whether the program goals and project objectives 
are being <met the Secretary should require project annual performance 
reports to Include Information on the academic performance of 
participating Students* and the status of all proposal objectives 
whether accomplished or not. 



O 23 ^0 



ERIC 



APPENDIX IV 



Atf&NDIX IV 



A View SSDS Project Performance Report 1s now being developed for reporting at, 
the end of the 1982-83 program year. The new report will require specific 
data on the academic performance and progress of project participants. 
This new reporting Instrument will serve as both a project monitoring 
device and a means for aggregating overall student performance, retention, 
and graduation data for the purpose of assessing program goals v 
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